Deep Learning and Hierarchal Generative Models

In this paper we propose a new prism for studying deep learning motivated by connections between deep learning and evolution. Our main contributions are: • We introduce of a sequence of increasingly complex hierarchical generative models which interpolate between standard Markov models on trees (phylogenetic models) and deep learning models. • Formal definitions of classes of algorithms that are not deep. • Rigorous proofs showing that such classes are information theoretically much weaker than optimal “deep” learning algorithms. In our models, deep learning is performed efficiently and proven to classify correctly with high probability. All of the models and results are in the semi-supervised setting. Some open problems and future directions are presented.